<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<!--[if IE]><meta http-equiv="X-UA-Compatible" content="IE=edge"><![endif]-->
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="Asciidoctor 1.5.8">
<title>Introduction</title>
<link rel="stylesheet" href="css/spring.css">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
</head>
<body class="article toc2 toc-left">
<div id="header">
<div id="toc" class="toc2">
<div id="toctitle">Table of Contents</div>
<ul class="sectlevel1">
<li><a href="#introduction">Introduction</a>
<ul class="sectlevel2">
<li><a href="#five-second-introduction">Five-second Introduction</a></li>
<li><a href="#five-minute-introduction">Five-minute Introduction</a></li>
<li><a href="#project-crawler">Project Crawler</a></li>
</ul>
</li>
</ul>
</div>
</div>
<div id="content">
<div class="sect1">
<h2 id="introduction"><a class="link" href="#introduction">Introduction</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>This section describes how Concourse works with Cloud Pipelines.</p>
</div>
<div class="admonitionblock important">
<table>
<tr>
<td class="icon">
<i class="fa icon-important" title="Important"></i>
</td>
<td class="content">
You do not need to use all the pieces of Cloud Pipelines. You
can (and should) gradually migrate your applications to use those pieces of
Cloud Pipelines that you think best suit your needs.
</td>
</tr>
</table>
</div>
<div class="sect2">
<h3 id="five-second-introduction"><a class="link" href="#five-second-introduction">Five-second Introduction</a></h3>
<div class="paragraph">
<p>Cloud Pipelines Concourse provides setup for Concourse that creates jobs and pipelines.
The pipelines and jobs use the scripts defined in
<a href="https://github.com/CloudPipelines/scripts">Cloud Pipelines Scripts</a> repo.</p>
</div>
</div>
<div class="sect2">
<h3 id="five-minute-introduction"><a class="link" href="#five-minute-introduction">Five-minute Introduction</a></h3>
<div class="paragraph">
<p>In these sections you will learn how exactly Cloud Pipelines Concourse integrates
with <a href="https://github.com/CloudPipelines/scripts">Cloud Pipelines Scripts</a> and how
you can setup deployment pipelines for each project.</p>
</div>
<div class="sect3">
<h4 id="how-to-use-it"><a class="link" href="#how-to-use-it">How to Use It</a></h4>
<div class="paragraph">
<p>The suggested approach is to use the <a href="#project-crawler">Project Crawler</a> approach, where
we scan your organization for projects and create deployment pipeline for each.</p>
</div>
<div class="paragraph">
<p>Another approach is to add the contents of each project and alter it to suit your needs.</p>
</div>
</div>
<div class="sect3">
<h4 id="how-it-works"><a class="link" href="#how-it-works">How It Works</a></h4>
<div class="paragraph">
<p>As the following image shows, Cloud Pipelines contains logic to generate a
pipeline and the runtime to execute pipeline steps.</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://raw.githubusercontent.com/CloudPipelines/concourse/master/docs/images/intro/how.png" alt="how">
</div>
<div class="title">Figure 1. How Cloud Pipelines works</div>
</div>
<div class="paragraph">
<p>Once a pipeline is created the jobs are ran, they clone or download Cloud Pipelines
code to run each step. Those steps run functions that are
defined in <a href="https://github.com/CloudPipelines/scripts">Cloud Pipelines Scripts</a>.</p>
</div>
<div class="paragraph">
<p>Cloud Pipelines performs steps to guess what kind of a project your
repository is (e.g. JVM or PHP) and what framework it uses (Maven or Gradle), and it
can deploy your application to a cloud (e.g. Cloud Foundry or Kubernetes)</p>
</div>
<div class="paragraph">
<p>Cloud Pipelines Concourse contains bash scripts that are required at runtime of
execution of a Concourse pipeline. If you want to read its documentation, it&#8217;s
available under <a href="https://github.com/{org}/{repo}/blob/{branch}/src/tasks/README.adoc"><code>src/tasks/README.adoc</code></a> file of Cloud Pipelines repository.</p>
</div>
<div class="paragraph">
<p>You can <a href="BASH_SCRIPTS.html">click here to go the separate subpage containing that documentation</a>.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="project-crawler"><a class="link" href="#project-crawler">Project Crawler</a></h3>
<div class="paragraph">
<p>Cloud Pipelines Concourse comes with a <code>crawler.groovy</code> file that allows to
go through the repositories in an organization in an SCM server (e.g. Github, Gitlab, Bitbucket)
and for each repo it will create a pipeline in Concourse.</p>
</div>
<div class="paragraph">
<p>We use the <a href="https://github.com/CloudPipelines/project-crawler">Project Crawler</a>
library, which can:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Fetch all projects for a given organization.</p>
</li>
<li>
<p>Fetch contents of a file for a given repository.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>For your convenience we&#8217;re providing you with a script under <code>src/pipeline/set_pipeline_crawler.sh</code>
that:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Verifies if Groovy is installed. If not, fetches it and unpacks to <code>build/groovy</code></p>
</li>
<li>
<p>Defines all the required environment variables</p>
</li>
<li>
<p>Runs the <code>crawler.groovy</code> script</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The following diagram depicts this situation:</p>
</div>
<div class="imageblock">
<div class="content">
<img src="crawler.png" alt="crawler" width="1053" height="590">
</div>
</div>
<div class="paragraph">
<p>Project Crawler supports repositories stored at Github, Gitlab, and Bitbucket.
You can also register your own implementation. See the
<a href="https://github.com/CloudPipelines/project-crawler">Project Crawler</a> repository for more information.</p>
</div>
<div class="paragraph">
<p>Below you can find the environment variables required by the Bash script</p>
</div>
<div class="exampleblock">
<div class="content">
<div class="listingblock">
<div class="content">
<pre class="highlightjs highlight"><code class="language-bash hljs" data-lang="bash"># root URL of the SCM server's API
export CRAWLER_ROOTURL="${CRAWLER_ROOTURL:-https://github.com}"
# username to connect to the API
export CRAWLER_USERNAME="${CRAWLER_USERNAME:-}"
# password to connect to the API
export CRAWLER_PASSWORD="${CRAWLER_PASSWORD:-}"
# token to connect to the API
export CRAWLER_TOKEN="${CRAWLER_TOKEN:-}"
# pattern to exclude certain projects
export CRAWLER_REPOPROJECTSEXCLUDEPATTERN="${CRAWLER_REPOPROJECTSEXCLUDEPATTERN:-}"
# type of SCM repo server (GITHUB, GITLAB, BITBUCKET, OTHER). Can be resolved from URL
export CRAWLER_REPOTYPE="${CRAWLER_REPOTYPE:-}"
# name of the pipeline descriptor
export CRAWLER_ORG="${CRAWLER_ORG:-}"
# organization / project name
export CRAWLER_PIPELINEDESCRIPTOR="${CRAWLER_PIPELINEDESCRIPTOR:-}"
# alias to be used by FLY CLI to set the pipeline
export CRAWLER_ALIAS="${CRAWLER_ALIAS:-}"
# credentials file to be used by FLY CLI to set the pipeline
export CRAWLER_CREDENTIALS="${CRAWLER_CREDENTIALS:-}"</code></pre>
</div>
</div>
</div>
</div>
<div class="paragraph">
<p>Here you can find the description of the Groovy script</p>
</div>
<div class="exampleblock">
<div class="content">
<div class="listingblock">
<div class="content">
<pre class="highlightjs highlight"><code class="language-bash hljs" data-lang="bash">	The arguments for this script will be taken in the order:
	Arguments, System Properties, Environment Variables

	Arguments:
	0 - rootUrl - root URL of the SCM server's API
	1 - username - username to connect to the API
	2 - password - password to connect to the API
	3 - token - token to connect to the API
	4 - repoProjectsExcludePattern - pattern to exclude certain projects
	5 - repoType - type of SCM repo server (GITHUB, GITLAB, BITBUCKET, OTHER)
	6 - org - organization / project name
	7 - pipelineDescriptor - name of the pipeline descriptor
	8 - alias - alias to be used by FLY CLI to set the pipeline
	9 - credentials - credentials file to be used by FLY CLI to set the pipeline

	System props with the name equal to arguments (e.g. rootUrl)
	Environment variables are upper case arguments with CRAWLER_ prefix (e.g. CRAWLER_ROOTURL)

	E.g. of calling the script

	#!/bin/bash

	set -o errexit
	set -o errtrace
	set -o pipefail

	export CRAWLER_ROOTURL="https://github.com"
	export CRAWLER_USERNAME="username"
	export CRAWLER_PASSWORD="password"
	export CRAWLER_TOKEN=""
	export CRAWLER_REPOPROJECTSEXCLUDEPATTERN=""
	export CRAWLER_REPOTYPE=""
	export CRAWLER_ORG="my-org"
	export CRAWLER_PIPELINEDESCRIPTOR="my-pipelines.yml"
	export CRAWLER_ALIAS="alias"
	export CRAWLER_CREDENTIALS="credentials-cf.yml"

	./set_pipeline_crawler.sh</code></pre>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<link rel="stylesheet" href="js/highlight/styles/atom-one-dark-reasonable.min.css">
<script src="js/highlight/highlight.min.js"></script>
<script>hljs.initHighlighting()</script>
</body>
</html>